Skip to content

Evidence search for claims with no source, and a local fallback judge - #9

Merged
Yigtwxx merged 7 commits into
mainfrom
feat/evidence-search
Sep 29, 2026
Merged

Yigtwxx merged 7 commits into
mainfrom
feat/evidence-search

Conversation

@Yigtwxx

@Yigtwxx Yigtwxx commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

What this changes

Text that cites nothing is searched on the web: Tavily or SearXNG, experimental, opt in by setting a provider. The pages found are read and scored like cited sources.

When Groq is rate-limited or stops answering, the judge switches at once to an installed local qwen3.5:9b and says so.

Why

A pasted post or plain claim with no link used to end in a parse error, and there was nothing to check it against (OPEN-ITEMS §17.1). Groq's free tier also runs out, and the judge then went silent.

In more detail:

  • Evidence search.
    • Up to 5 check-worthy sentences are searched, 3 pages each.
    • A search snippet is never used as evidence.
    • Findings are tagged FOUND BY PROOFPATH (not cited by the author).
    • An empty search is reported as NO EVIDENCE FOUND (searched), never REFUTED.
    • A run that searched only some of its claims exits 1.
  • Translation. A non-English claim is checked as its judge-made English translation, with a checked as: line. With no judge it is UNVERIFIED (language not supported).
  • Local fallback judge.
    • No retry and no wait: the failed request is re-sent to the local model.
    • The switch is announced on stderr, in the TUI above the prompt, and in the report.
    • The next run starts on Groq again.
    • The model is never pulled.
  • Fixed. A bare DOI or arXiv URL is never reported as a GHOST REFERENCE. A v5 cache migration drops ghost results cached under the old rule.
  • Measured. On AVeriTeC dev, 100 claims, the product's own search path scores 0.404 (three-way) against a 0.708 majority baseline; the gold source URLs scored 0.270. The §17.1 gate is not met, so the feature stays experimental.

Checklist

  • ruff check and ruff format --check are clean
  • pytest passes
  • Type annotations on every new function signature
  • No network access in unit tests
  • If a new reported state was added, spec section 15 is updated too
  • This change does not make the tool sound more confident than its evidence (experimental label, below-baseline score shown in every searched report)

🤖 Generated with Claude Code

https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux

Yigtwxx and others added 4 commits September 28, 2026 16:49
A pasted post or plain text with no link, marker or bibliography used to
end in a parse error. It is now searched: up to five check-worthy
sentences go to Tavily or a SearXNG instance, and the pages found are
read and scored through the usual fetch and NLI path. Each finding is
marked "found by proofpath", and an empty search is reported as NO
EVIDENCE FOUND (searched), never as REFUTED. A judge writes the queries
and translates non-English claims; when it is limited, the sentence
itself is searched and the run says so.

The judge now falls back from Groq to a local qwen3.5:9b in Ollama the
moment Groq is rate-limited or down. The request that failed is sent
again to the local model, the switch is announced (stderr, TUI line,
report), and the next run starts on Groq again. Nothing is ever pulled.

Also: --no-search, the search section in /config, footer lines, and an
AVeriTeC --search eval mode that excludes the claim's own fact-check site.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
The README gains a section on searching claims with no source, a row in
the What it reads table, and a note on the judge's fallback to a local
qwen3.5:9b. The landing page's reads list and the X hint name the new
path. The AVeriTeC score is a marker until the corrected eval runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
A page the search found through a DOI or arXiv URL was reported as a
GHOST REFERENCE, because the resolver compared a title against a URL.
Found pages now go straight to the web reader, and an entry that is only
an identifier is never called a ghost: it resolves, or it is AMBIGUOUS.
A v5 cache migration drops GHOST resolutions cached under the old rule.

The English detector flagged ordinary English ("MIT", "et al.", "Los
Angeles") as foreign, then missed short Spanish, French and German
sentences. It now ignores capitalised tokens, counts accented lowercase
words and a fixed set of function words, and a judge's translation is
used unless the original is certainly English.

A run that hit the claim cap now exits 1 even when every searched claim
was supported, so a partly searched run cannot look clean. The AVeriTeC
eval measures the product's own search path, and pages proofpath found
on its own are fetched without the contact address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
The product's own search path reads the pages it finds for 100 AVeriTeC
dev claims and scores 0.404 three-way, against a 0.708 majority
baseline. That is better than the 0.270 the gold source URLs gave, but
the gate of spec §17.1 is not met, so the feature stays experimental.
The run from 2026-09-28 used a path the final review replaced and is
dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
@vercel

vercel Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
proofpath Ignored Ignored Preview Sep 29, 2026 9:07am UTC

Yigtwxx and others added 2 commits September 29, 2026 11:47
The PR checklist asks for every new reported state to be in spec §15;
the four from the evidence-search spec were only there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
The README's state table and the landing page's state list, X row and
judge paragraph still described the tool before this branch: no search
states, and a judge that went quiet when its provider stopped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
When the judge's API gives out and no local model can take over, the
run now says how to add one: install Ollama and pull qwen3.5:9b, or
only pull the model when Ollama is already there. It goes out once per
run on the switch notice's channels, and never when the user turned the
fallback off. Nothing is installed; it is only a line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
@Yigtwxx
Yigtwxx merged commit 704cef1 into main Sep 29, 2026
9 checks passed
@Yigtwxx
Yigtwxx deleted the feat/evidence-search branch September 30, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant